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ABSTRACT 

Unmanned  underwater  vehicles  arc  becoming  an  increasingly  important  platform  in  oceanographic  research  and 
operational  oceanography,  where  continuous  in  situ  sampling  throughout  the  water  column  is  essential  to  understanding 
the  ocean  circulation  and  related  biological,  chemical,  and  optical  activity.  The  latter  directly  affects  field  operations  and 
remote  sensing  capabilities  from  space.  A  unified  approach  is  necessary  for  data  quality  control  (QC),  access,  and 
storage,  considering  the  vast  amount  of  data  collected  from  gliders  continuously  deployed  across  large  areas  and  over 
long  durations.  The  Binary  Universal  Form  for  the  Representation  of  meteorological  data  (BUFR)  maintained  by  the 
World  Meteorological  Organization  (WMO)  is  adapted  to  include  physical  and  optical  parameters  from  a  variety  of 
sensor  suites  onboard  underwater  vehicles.  The  provisional  BUFR  template  and  related  BUFR  descriptors  and  table 
entries  have  been  developed  by  the  U.S.  Navy  for  ocean  glider  profile  data  and  QC  results.  Software  written  in 
FORTRAN  using  the  ECMWF  BUFRDC  library  has  been  implemented  to  perform  both  the  encoding  and  decoding  of 
BUFR  files  from  and  to  Network  Common  Data  Form  (NetCDF)  files.  This  presentation  also  discusses  data  collected 
from  sensors  on  gliders  deployed  both  in  deep  water  and  shallow  water  environments,  including  issues  specific  to  optical 
sensors  at  various  depths. 


INTRODUCTION 

One  of  the  key  obstacles  in  oceanographic  research  and  operational  oceanography  is  the  difficulty  associated  with  field 
sampling  and  measurements,  due  to  lack  of  easy  access  to  the  vast  areas  of  the  ocean,  that  cover  approximately  70%  of 
the  planet  surface.  Remote  sensing  techniques  (both  active  and  passive),  especially  those  from  space,  have  been  proven 
to  offer  synoptic  surface  coverage  with  adequate  accuracy,  when  sensors  are  calibrated  and  validated  correctly  with  help 
from  in  situ  measurements,  while  the  effects  of  atmosphere  are  also  estimated  correctly.  Even  so,  such  signals  arc 
heavily  weighted,  thus  biased,  towards  the  features  from  the  surface  layer.  Ocean  circulation  models  offer  much  needed 
3-dimentionality  to  the  mix,  allowing  the  described  features  extending  beyond  the  surface  layers,  on  top  of  the  2D 
synoptic  coverage.  With  the  initial  boundary  conditions  properly  determined,  which  can  be  obtained  from  in  situ 
measurements  and  satellite  observations,  the  models  allow  the  4fh  dimensionality,  or  forecast,  be  included  in  the 
framework. 

Traditional  ship-based  ocean  sampling  techniques  are  still  widely  used  today.  Advances  have  been  made  in  great  stride 
recently  by  numerous  in  situ  observation  systems  on  moorings,  buoys,  floats,  flow-through  sensors,  unmanned 
underwater  vehicles  (UUVs),  such  as  gliders,  autonomous  underwater  vehicles  (AUVs),  remotely  operated  Vehicles 
(ROVs),  integrated  sensor  networks,  and  observatories.  These  are  vibrant  research  and  development  areas  and  generate 
the  most  accurate  three-dimensional  (3D)  data  available,  often  in  real-time,  and  are  less  affected  by  adverse  conditions. 
However,  spot  sampling  lacks  the  rapid,  broad  coverage  that  is  critical  in  high-level,  real-time  operational  decision 
making.  In  situ  observations  at  times  are  not  available  for  unsafe  or  denied-access  environments.  Remote  sensing 
techniques  can  be  used  to  fill  the  needs,  when  precise  protocols  are  in  place  to  maintain  data  coherence  and  accuracy. 
Further,  modem  defense  and  security  needs  demand  that  accurate  information  be  provided  when  and  where  it  is  needed 
(e.g.,  Battlespace  on  Demand,  BonD).  Ocean  sensing  must  not  only  provide  timely  and  accurate  data,  but  also  offer 
insights  regarding  overall  3D  and  future  -  or  forecasted-environmental  conditions.  The  combined  use  of  in  situ 
observations,  remotely  sensed  data,  and  physical  models  is  a  rapidly  evolving  field,  although  improved  assimilation  of 
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available  data  into  models  still  poses  a  challenge.  The  ability  to  sense,  integrate,  and  predict  is  vital  in  establishing  a  true 
real-time,  four-dimensional  (4D)  cube  of  verified  and  validated  information  for  ocean  nowcast  and  forecast,  as  shown  in 
Figure  1,  in  terms  of  Tactical  Ocean  Data  System  (TODS).  We  see  that  glider  observations  provide  critical  input  to  one 
of  the  key  elements  in  BonD  tier  0  structure. 

Tactical  Ocean  Data  System 
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Figure  1 .  Tactical  Ocean  Data  System  (TODS)  component  chart. 

Glider  observations  provide  inputs  that  are  not  only  limited  to  localized  or  spot  events,  but  are  also  capable  of  long- 
range,  long-term,  in-depth  throughout  the  water  column.  One  good  example  is  the  recent  7-month  voyage  by  a  Rutgers 
University  Slocum  glider  crossing  the  Atlantic  Ocean,  sampling  the  ocean  structures  along  the  way.  The  vast  data  stream 
generated  by  gliders  contains  a  wealth  of  much  needed  information  for  topics  discussed  above.  Automated  QC  and 
analysis  of  these  data  are  the  only  way  for  the  timely  assimilation  to  models  and  the  framework  above. 

This  paper  describes  the  capabilities  of  glider  optical  measurements  at  the  Naval  Research  Lab  (NRL)  and  the  Naval 
Oceanographic  Office  (NAVOCEANO),  and  efforts  associated  with  the  development  of  automated  optics  QC  processes 
and  storage  format. 


GLIDER  OPTICS  MEASUREMENTS  AND  PROCESSING 

To  answer  the  needs  of  Mine  Warfare  (M1W),  Anti-submarine  Warfare  (ASW),  safety  of  navigation,  monitoring  global 
climate  change,  and  battlespace  environment  sensing,  optical  sensors  have  been  fitted  and  tested  on  different  types  of 
gliders,  and  have  proven  successful  in  assessing  optical  conditions  in  a  variety  of  water  masses.  These  sensors  provide 
inputs  for  optimizing  ocean  optical  models  used  to  generate  and  forecast  electro-optical  identification  (EOID) 
performance  surfaces,  diver  visibility  and  asset  vulnerability  products,  which  can  aid  in  tactical  decision  making  during 
fleet  operations  (Figure  1). 
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Table  1.  NRL/NAVOCEANO  glider  optical  parameters  table. 


An  automated  QC  process  for  optical  measurements  from  gliders  is  designed  and  implemented  as  part  of  the  Local 
Automated  Glider  Editing  Routine  (LAGER)  [1][2].  The  optieal  algorithms  are  used  to  proeess  data  from  available 
glider  optieal  sensors  (Table  1),  including  instruments  that  measure  the  beam  attenuation  coefficient  (c),  the  total 
scattering  coefficient  (/)),  the  baekseatter  coefficient  ( bh ),  the  fluoreseenee  returns  and  thus  derived  concentrations  of 
chlorophyll,  phycoerythrin ,  and  colored  dissolved  organic  matter  (CDOM),  downwclling  irradianee  (£</), 
phytosynthetically  available  radiation  (PAR),  and  other  derived  optieal  parameters,  sueh  as  diver  visibility  (Table  1).  All 
current  optieal  sensors,  including  the  ECO  series  from  Wetlabs  (bb2f,  bb2slo,  fl3,  SAM,  BAM,  AUVb,  ECO-PAR),  and 
OCR504/7  from  Satlantie,  are  included  in  the  current  operational  version,  with  flexibility  built  in  to  allow  for  expansion 
to  meet  future  needs.  Additionally,  Table  1  outlines  the  typical  range,  resolution,  bandwidth,  and  channel  limits  of  the 
optieal  sensors,  which  are  used  in  the  LAGER  Opties  routines  and  will  be  diseussed  in  later  sections.  The  data  flow  can 
be  viewed  as  a  5-step  proeess,  with  the  inclusion  of  opties,  as  shown  in  Figure  2.  Notice  that  the  proeess  ean  be  enabled 
sueh  that  all  data  will  (1)  be  sent  to  manual  editing,  (2)  undergo  physical  parameter  processing  only,  or  (3)  undergo  both 
physical  and  optieal  parameters  processing. 

Glider  data  ingest  module  reads  in  real-time  raw  data  at  the  NAVOCEANO  Glider  Operation  Center  (GOC)  and 
converts  to  Network  Common  Data  Form  (NetCDF)  format.  Automated  QC  is  the  key  step  in  LAGER  Opties.  It  uses 
flags  to  classify  different  types  of  erroneous  data  points,  and  the  combined  quality  of  the  flags  determines  whether 
Manual  User  GUI  (MUG)  is  needed  for  closer  inspection.  The  resulting  data  are  then  sent  to  the  database  in  a  binary 
universal  format  for  representing  meteorological  data  (BUFR)  format  required  by  the  real-time  data  handling  system 
(RTDHS)  at  NAVOCEANO. 

The  automated  optieal  QC  program  structure  is  shown  in  Figure  3.  The  detailed  description  of  eaeh  module  ean  be  found 
in  [2].  Briefly,  QC  tests  for  optieal  variables  implemented  in  LAGER,  follow  certain  algorithms  and  conventions  for 
detecting  and  flagging  bad  data  that  were  developed  for  processing  the  physical  variables,  sueh  as  the  temperature  (T), 
and  the  salinity  (S).  However,  because  of  the  multiplicity  of  opties  variable  types  and  frequency  bands,  a  2D  variable 
array  structure  is  used  in  contrast  to  the  ID  vector  approach  used  individually  for  T  and  S.  The  reader  is  referred  to  [1] 
for  details  of  the  physical  data  QC  implementation.  Two  of  the  physical  QC  algorithms  implemented  in  LAGER  are  used 
with  minor  adaptations  for  the  Opties  QC  (OQC).  These  arc  the  global  bounds  check  and  the  spike  test.  The  latter  defines 
a  spike  to  be  a  single  datum  departing  significantly  from  its  neighboring  values  (single-point  spike).  It  has  the  additional 
feature  of  being  able  to  detect  such  a  spike  in  the  presence  of  a  gradient  (as  a  function  of  depth)  of  the  relevant 
hydrographic  or  optical  variable.  Since  segments  of  anomalous  optical  data  may  span  several  data  points,  a  method  to 
detect  these  eases,  in  the  form  of  a  running  standard  deviation  filter,  is  also  included  in  the  OQC.  This  filter  finds  and 
flags  data  that  depart  significantly  in  value  from  a  local  mean  computed  inside  a  ‘running’  window  (depth  interval)  that 
is  moved  through  the  entire  sampled  depth  range  of  the  profile.  In  addition  to  the  variable  value  range  eheek,  the  depth  of 
eaeh  sample  point  is  also  cheeked  and  ‘chopped’  short,  if  neeessary,  to  eliminate  values  that  appear  to  lie  above,  or  too 
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close  to  the  surface,  where  processes  such  as  wind  waves  and  bubbles  could  make  optical  measurements,  such  as 
baekseattering  and  downwelling  irradianee,  cither  invalid  or  difficult  to  interpret  reliably.  If  this  depth  is  shallower  than  a 
given  depth  (default  1  m)  or  is  negative,  the  variable  value  is  flagged  accordingly.  A  constant  profile  check  is  also  done 
to  determine  if  the  values  for  a  particular  optics  variable  are  constant  throughout  the  sampled  depth  range,  which  was 
proven  critical  in  assessing  real-time  eleetro-optieal  sensor  performance,  such  as  those  during  the  RIMPAC  2008 
exercise  [3].  This  might  occur  for  a  variety  of  reasons,  such  as  instrument  sampling  faults,  or  sensor  contamination, 
which  suggest  the  data  are  invalid. 


Utilities 

Post  procoss 


Physical  -Optical 
Optimization 
•  And  3d  Volume 

'  t 


Figure  2.  Data  flow  chart  of  LAGER  Optics. 

In  the  present  LAGER  opties  algorithm,  all  the  OQC  tests  are  independent  of  geographic  location  and  time  of  year,  and 
the  tests  are  also  independent  of  depth,  except  in  eases  where  different  critical  test  values  are  used  in  two  different  depth 
ranges.  The  universal  character  of  the  tests  weakens  their  capability  to  detect  erroneous  anomalies  in  specific  regions. 
However,  most  of  the  parameters  used  to  configure  the  OQC  algorithms  are  stored  in  information  files,  and  may  be 
edited  to  adapt  them  manually,  if  neecssary,  to  specific  regions,  or  seasons.  A  future  version  of  LAGER  will  include 
capabilities  to  adapt  these  parameters  to  specific  regions  automatically,  based  on  a  regional  climatology  or  historical 
data,  or  alternatively,  specific  regimes,  such  as  shallow  or  deep  ocean  settings,  or  proximity  to  the  coast,  and  relationship 
to  other  physical  parameters.  This  is  particularly  important  for  optical  data,  for  which  the  dynamic  range  varies  widely, 
depending  upon  the  prevailing  oceanographic  setting  or  conditions.  In  highly  productive,  nutrient-rieh  regimes  and/or 
close  to  estuarine  freshwater  sources,  values  of  Chlorophyll-a  and  CDOM  at  the  surface,  or  in  the  pyenoeline,  may  reach 
highly-elevated  values,  and  may  exhibit  very  steep  property  gradients,  that  are  not  observed  in  deeper  water. 

For  the  physical  variables,  several  glider-speeifie  tests  have  been  added  to  the  OQC  to  detect  and  flag  specific  known 
types  of  bad  behavior  exhibited  by  either  specific  kind  of  gliders  or  by  all  types  of  gliders.  In  most  eases,  the  glider- 
specific  tests  are  functions  of  the  vertical  velocity  of  the  glider  that  is  employed  as  a  substitute  for  the  total  speed  of  the 
glider  through  the  water,  which  is  difficult  to  determine.  There  arc  some  early  indications  that  similar  provisions  will 
need  to  be  made  for  the  optical  data.  However,  insufficient  data  have  been  accumulated  to  clearly  establish  the  need  for 
such  corrections  and  to  reliably  specify  a  remedial  strategy. 
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Optieal  error  flags  are  initially  set  to  zero  at  each  depth.  Tests  are  performed  on  observed  temperature  and  salinity  values 
prior  to  any  tests  performed  on  optieal  values.  If  a  test  fails  at  a  specified  depth,  then  the  corresponding  flag  is  set  to  the 
failure  flag  value  for  that  test  at  that  depth.  The  flag  values  for  corresponding  test  are  listed  below: 


Flag  name 


assigned  value 


Global  bounds  check  2 

Bb  buddy  eheck  4 
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Figure  3.  LAGER  Optics  VI. 0  data  quality  check  flow  chart. 
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The  LAGER  OQC  sends  a  multi-sensor  multiple-band  optics  file  for  further  evaluation  in  the  manual  QC  step  if,  for  any 
of  the  individual  profiles  in  the  file,  the  number  of  profile  observations  flagged  as  bad  is  greater  than  a  given  percentage 
(default  15%  for  the  current  configuration)  of  the  total  number  of  observations  of  that  profile.  It  is  planned  to  examine 
additional  criteria  to  determine  if  the  quality  of  the  profiles  on  the  basis  of  the  temperature  and  salinity  or  optics  variable 
values.  (See  [I]  for  details  eoneeming  the  corresponding  criteria  for  the  physical  data).  Henee,  a  file  may  be  sent  to 
manual  QC  because  either  or  both  of  temperature  or  salinity  require  editing  or  deletion,  or  because  at  least  one  of  the 
optical  profiles  requires  visual  inspection.  In  addition,  an  individual  T,  S,  or  optical  variable  profile  may  be  flagged  for 
deletion  if  the  number  of  bad  data,  or  gaps,  are  excessive.  A  needs  manual  editing  flag  is  assigned  in  eaeh  file  to  eaeh 
ascending  or  descending  profile  (eaeh  value  summarizes  quality  of  the  corresponding  optics  variable  profile). 


GLIDER  OPTICAL  TABLE  AND  ADAPTED  BUFR 

To  meet  automated  data  QC,  downstream  model  requirements  and  archive  needs,  BUFR  format  is  adapted  from 
meteorological  research  to  be  used  at  the  NAVOCEANO  GOC.  All  glider  sensor  outputs  including  temperature,  salinity 
and  various  optical  parameters  are  examined  and  converted  in  BUFR  data  tables.  Details  of  such  tables  ean  be  found  in 
[4].  BUFR  has  been  used  since  1989  for  data  archive  and  maintained  by  the  World  Meteorological  Organization 
(WMO),  which  is  described  in  details  at  the  following  website: 

httn://www.wrmo.ch'pages/proe/wfwrwAVMOCodes/OperationalCodes.html.  Briefly,  a  BUFR  message  contains  6 
sections  (0-5)  to  store  header  and  data.  Section  3  contains  a  short  header  followed  by  data  descriptors  which  defines  the 
form  and  content  of  the  bit-stream  in  Section  4.  The  sequence  of  descriptors  in  Section  3  could  be  viewed  as  the  template 
of  the  BUFR  message.  This  template  contains  the  information  needed  to  describe  the  structure  of  the  data  values 
embedded  in  the  matching  bit-stream  in  the  following  section  (Section  4).  The  template  is  to  be  interpreted  in  a  step-by- 
step,  algorithm-like  manner.  Given  a  set  of  BUFR  messages,  the  values  contained  in  the  bit  stream  in  Section  4  may 
differ  from  one  message  to  the  next  corresponding  to  different  glider  profiles,  but  their  ordering  and  structure  will  be 
kept  predictable  if  the  template  provided  in  Section  3  remains  unchanged.  The  NRL/NAVOCEANO  optics  table  (Table 
1 )  has  been  designed  and  incorporated  into  NAVOCEANO  BUFR  tables,  with  an  assigned  Class  60  descriptor  (Table  2). 
The  optical  variables  QC  flags  corresponding  to  Table  I  are  shown  in  Table  3. 
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Table  2.  BUFR  optics  Table  (class  60),  adapted  from  the  optics  parameters  listed  in  Table  1 . 

Traditional  optical  variable  names  and  units  are  used.  Typical  values  listed  are  adapted  from  open  ocean  or  clear  water 
environments.  Bandwidth  parameters  are  from  current  or  past  instruments  tested  on  a  variety  of  ocean  gliders,  with 
names  denoted  from  raw  data  stream.  The  spectral  information  is  not  listed  in  this  table  and  is  subject  to  change, 
depending  on  individual  sensors  onboard  gliders.  Multiple  wavelengths  of  certain  sensors  (e.g.  bb3slo)  should  be 
expected  and  have  been  implemented  in  the  BUFR  table  by  means  of  repetition. 
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The  above  automated  QC  routines  are  in  place  and  have  been  tested  under  different  conditions,  using  data  from  different 
types  of  gliders.  The  results  show  [3]  that  under  the  most  complicated  situations  involving  complexities  associated  with 
data  transmission  as  well  as  environmental  variability,  as  much  as  30%  of  profiles  require  manual  examination  by 
operators.  The  failure  rate  in  detection  of  bad  profiles  is  less  than  0.4%  of  the  profiles  examined,  which  is  associated 
with  sparseness  of  the  data,  and  single  point  spikes  likely  due  to  subsurface  optical  layers. 
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Table  3.  Quality  control  class  (63).  See  text  for  details. 


SUMMARY 

Glider  optical  measurements  provide  critical  input  to  establish  an  accurate  optical  ocean  sensing  and  forecasting  system, 
that  fuses  observations  from  satellite  remote  sensing  sensors  and  outputs  from  ocean  circulation  models.  Together,  a  4D 
verified  data  cube  can  be  utilized  in  real-time  operational  decision  making.  An  automated  glider  optics  QC  package  is 
needed  to  handle  the  large  volume  of  input  data,  while  it  outputs  much  needed  information  for  downstream  optical 
products,  including  3D  structure  optimization  and  performance  surface  predictions  for  electro-optical  identification 
(EOID)  sensors.  Extension  testing  has  been  implemented  and  results  show  a  low  level  failure  rate,  which  can  be  further 
improved  once  current  development  algorithms  are  incorporated. 
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